What changes were proposed in this pull request?

Implements RANGER-5655: a dynamic unified ingestor registry for Ranger audit-ingestor so operators can change Kafka partition routing and per-repo service allowlists at runtime — without restarting ingestor pods.

The registry is a versioned JSON document in Kafka topic ranger_audit_partition_plan (1 partition, compacted). All ingestor replicas converge via PartitionPlanWatcher; AuditPartitioner routes on the hot path from in-memory state only.

Feature flag (default off): ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false

Problem

Solution (this PR update)

Simplified REST control plane — three endpoints only:

Removed (consolidated above): PATCH /api/audit/partition-plan, POST /api/audit/partition-plan/services, separate promote-only / scale-only flows.

New request models: OnboardPlugin, UpdatePlugin. Service entries stored with optional pluginId for repo→plugin ownership.

Code changes in this commit (REST simplification slice)

How was this patch tested?

Unit tests + quality gates

mvn verify -pl audit-server/audit-ingestor -Drat.skip=true \
  -Dtest='PartitionPlan*Test,ServiceAllowlist*Test,AuthToLocalRuleComposerTest'

Focused run:

mvn test -pl audit-server/audit-ingestor \
  -Dtest='PartitionPlan*Test,ServiceAllowlist*Test' -Drat.skip=true

Manual testing (local Docker audit lab)

Manual validation used a local Docker Compose audit environment that mirrors a production-style Ranger audit deployment: Kerberos (KDC + plugin keytabs), Kafka with both the audit data topic (ranger_audits) and the compacted registry topic (ranger_audit_partition_plan), a running audit-ingestor instance on port 7081, Solr (with audit dispatcher), Postgres-backed Ranger Admin, and real plugin containers for HDFS and Hive. All partition-plan REST calls used SPNEGO (Kerberos) as the ingestor HTTP service principal; plugin audit posts used each plugin’s own keytab.

The ingestor was rebuilt with this branch’s code (including mandatory services validation on onboard) before running the scenarios below.

1. Environment readiness

Before exercising the new API, the lab was brought to a healthy state: ingestor health endpoint returned 200, Kafka was reachable, the plan watcher was active after enabling dynamic mode, and GET /api/audit/partition-plan returned a coherent plan JSON (version, plugins, buffer, services, topicPartitionCount matching the live ranger_audits partition count).

2. Static mode unchanged (feature flag off)

With ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false (default):

• GET /api/audit/partition-plan returned 503 — partition-plan admin API correctly disabled.
• GET /api/audit/health still returned 200.
• Normal plugin audit delivery (HDFS smoke, Solr indexing) continued to work.

This confirms existing deployments are unaffected when the flag stays off.

3. Enabling dynamic mode and reading the registry

Dynamic mode was turned on (dynamic.enabled=true) with a fresh or reset plan topic where appropriate. After ingestor restart:

• Kafka showed ranger_audit_partition_plan with one partition and compacted cleanup policy.
• GET /api/audit/partition-plan returned 200 with version ≥ 1, populated services from XML bootstrap, and topicPartitionCount equal to kafka-topics --describe ranger_audits.
• Ingestor logs confirmed PartitionPlanWatcher started and the partitioner loaded the in-memory plan.

4. Simplified REST API — onboard, validation, scale

All mutations used expectedVersion from the preceding GET.

Negative validation (new behavior)

• POST /api/audit/partition-plan/plugins with pluginId, partitionCount, and expectedVersion but omitting services → 400 Bad Request with message indicating services are required. This was the primary regression guard for the API consolidation.

Successful onboard

• Onboarded a buffer-only plugin (e.g. storm or ambari) in a single call with a non-empty services map (repo → allowedUsers). Response 200; plan version incremented; plugin appeared under plugins with dedicated partition IDs taken from the buffer (or tail-grown when needed); corresponding repo entries appeared under services.

Multi-repo onboard in one version bump

• Onboarded trino with two repos in one POST (dev_trino and dev_trino2, each with its own allowedUsers) → 200; both repos present in services with pluginId ownership tagged to trino.

Optimistic locking

• Repeated onboard with a stale expectedVersion → 409 Conflict with current plan in the response body.
• Attempted to onboard hdfs again when it already had dedicated partitions → 400 (conflicting state).

Scale after onboard

• PATCH /api/audit/partition-plan/plugins/{pluginId} with additionalPartitions → 200; tail partition IDs appended append-only; ranger_audits grown via AdminClient when required; subsequent GET showed stable version and layout.

Idempotency check

• After mutations, GET /api/audit/partition-plan without restart showed the same version and layout as the last successful write.

5. End-to-end plugin flows (allowlist + routing)

These tests prove the full path: registry onboard → allowlist enforcement → Kafka produce → correct partition assignment.

HDFS

1. Onboarded plugin hdfs with repo dev_hdfs and allowlist hdfs,nn via POST .../plugins (mandatory services).
2. From the Hadoop container, posted a test audit batch to POST /api/audit/access?serviceName=dev_hdfs&appId=hdfs using the hdfs Kerberos principal → 200; authenticatedUser mapped to short name hdfs.
3. Consumed the corresponding record from ranger_audits and verified the partition number was in the hdfs assignment list from the plan (not the buffer pool).
4. Optionally scaled hdfs with PATCH .../plugins/hdfs and repeated the access + Kafka partition check — routing still respected the updated plan.

Hive

1. Onboarded hiveServer2 with repo dev_hive and allowlist ["hive"] in the same onboard POST.
2. From the Hive container, posted audits with the hive principal → 200.
3. Kafka record landed on a partition in the hiveServer2 dedicated set (e.g. partitions [7, 8] after prior lab mutations).
4. Confirmed ingestor logs showed auth_to_local rules recomposed after the onboard (allowlist union updated).

HDFS already onboarded path

• Where hdfs was already present in the plan from an earlier run, the lab skipped re-onboard and verified allowlist + routing still held: access accepted, partition ∈ plan.

6. Allowlist behavior (authorization layer)

Separate from partition routing:

• A principal in services[repo].allowedUsers (after auth_to_local) → 200 on /access.
• After tightening allowlist via PATCH .../plugins/{pluginId} with updateServices to remove the principal → 403 on the same POST.
• Restoring the allowlist → 200 again.
• Posting audits claiming a different repo than the principal is allowed for → 403 (cross-repo denial).

This confirms the unified services map in the registry drives authorization without XML restart.

7. What did not change

8. Summary of manual test outcomes